Probabilistic Planning with Reduced Models

Markov decision processes (MDP) (Puterman 1994) offer a rich model that has been extensively used by the AI community for planning and learning under uncertainty. Some applications include planning for mobile robots, network management, optimizing software on mobile phones, and managing water levels of river reservoirs. MDPs have polynomial complexity in the size of the state space, but the state space itself is exponential in the description size. Therefore, algorithms that try to find complete optimal plans are often impractical. Developing effective ways to tackle this complexity barrier is a challenging research problem. Determinization-based algorithms for solving MDPs have gained popularity in recent years (Yoon et al. 2008; Teichteil-Königsbuch et al. 2010; Keyder and Geffner 2008), motivated by the surprising success of the FF-Replan solver (Yoon et al. 2007). The main idea is to generate a deterministic version of the underlying MDP and solve it using a classical deterministic planner, resulting in a partial plan for the original problem. When confronted by an unexpected state during plan execution, the planning process is repeated using the current state as the initial state. The advantage of this approach is its ability to quickly generate partial plans, particularly in intractable probabilistic domains. Despite their success, determinization-based algorithms have drawbacks because they consider action outcomes in isolation. This leads to an overly optimistic view of the domain and can result in plans arbitrarily worse than optimal. Furthermore, even when optimal plans could be obtained using isolated outcomes, it is not always clear, nor intuitive, which outcomes should be included in the determinization. In my work I introduce and study a more general paradigm in which the single-outcome variant of FF-Replan is just one extreme point on a spectrum of MDP reductions that differ from each other along two dimensions: (1) the number of outcomes per state-action pair that are fully accounted for in the reduced model, and (2) the number of occurrences of the remaining outcomes that are planned for in advance. Similar treatments of exceptional outcomes have been explored in fault-tolerant planning (Jensen et al. 2004; Domshlak 2013; Pineda et al. 2013).